Repository navigation
[datasets sql] Run the DuckDB CLI subprocess with UTF-8 encoding - #5020
Merged
Merged
Conversation
Wauplin
reviewed
Sep 28, 2026
Wauplin
reviewed
Sep 28, 2026
Co-authored-by: Lucain <lucainp@gmail.com>
Wauplin
approved these changes
Sep 28, 2026
Wauplin
left a comment
Collaborator
There was a problem hiding this comment.
Makes sense. Thanks for reporting and fixing 🤗
|
The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update. |
Contributor
|
This PR has been shipped as part of the v2.1.0 release. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #5019.
_DuckDBCliRelation.executeruns the DuckDB CLI withtext=Truebut noencoding, so the pipes use the locale encoding. DuckDB's stdin/stdout are UTF-8 on every platform, which on stock Windows (cp1252) means:SELECT chr(233) AS city→[{'city': 'é'}]UnicodeDecodeError),result.stdoutisNone, and the library crashes withAttributeError: 'NoneType' object has no attribute 'strip'_stdin_writewithUnicodeEncodeErrorOne-line fix: pass
encoding="utf-8"to thesubprocess.runcall. Verified on Windows 11 / cp1252 / DuckDB CLI v1.5.5 — all three cases return correct data (é,Á,中文) after the change. TheduckdbPython package path was never affected.Added
TestDuckDBCliRelationEncodingintests/test_cli.py: mocks the subprocess and asserts the UTF-8 encoding kwarg; fails onmain, passes with the fix.ruff check/ruff format --checkclean;pytest tests/test_cli.py -m "not production"→ 372 passed (the 2 failures also fail on a clean checkout ofmainin this environment).Note
Low Risk
Single-parameter change aligning subprocess text mode with DuckDB’s documented UTF-8 behavior; scope is limited to the CLI dataset SQL path.
Overview
Fixes incorrect decoding when
_DuckDBCliRelation.executetalks to the DuckDB CLI on locales where the default pipe encoding is not UTF-8 (e.g. Windows cp1252).The
subprocess.runcall now passesencoding="utf-8"alongsidetext=True, so stdin/stdout match DuckDB’s UTF-8 I/O. That avoids mojibake on non-ASCII result cells,UnicodeDecodeError/ empty stdout on some byte sequences, andUnicodeEncodeErrorwhen the SQL string contains non-ASCII text.Reviewed by Cursor Bugbot for commit 243947d. Bugbot is set up for automated code reviews on this repo. Configure here.